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IN THE UNITED STATES ELECTED OFFICE 
OF THE UNITED STATES PATENT AND TRADEMARK OFFICE 
UNDER THE PATENT COOPERATION TREATY-CHAPTER II 

" PRELIMINARY AMENDMENT " 


APPLICANT: Gustavo DECO et al. 

SERIAL NO.: EXAMINER: 
FILING DATE: ART UNIT: 

INTERNATIONAL APPLICATION NO.: PCT/DE99/01949 
INTERNATIONAL FILING DATE: 1 July 1 999 

INVENTION: METHOD FOR TRAINING A NEURAL NETWORK, 

METHOD FOR CLASSIFYING A SEQUENCE OF 
INPUT PARAMETERS USING A NEURAL NETWORK, 
NEURAL NETWORK AND ARRAY FOR TRAINING A 
NEURAL NETWORK 

Hon. Assistant Commissioner for Patents 
Box PCT 
Washington D.C. 20231 


SIR: 

Amend the above-identified international application before entry into the 
national stage before the U.S. Patent & Trademark Office under 35 U.S.C. §371 
as follows: 


IN THE SPECIFICATION 

An amended substitute specification is attached, and a marked up copy of 
the original specification showing the changes is also enclosed. 


IN THE CLAIMS 

Amend the claims as shown in the following replacement claims, the 
marked up version of which is attached following the marked up specification. 

We Claim: 

1 . A method for training a neural network that contains pulsed neurons, 

a) training the neural network for a first time span such that a discrimination 
value is maximized, as a result whereof a maximum first discrimination 
value is formed; 

b) forming the discrimination value dependent on pulses that are formed by 
the pulsed neurons within the first time span as well as on a training 
sequence of input quantities that are supplied to the neural network; 

c) implementing the following steps interactively: 
shortening the first time span to form a second time span, 
forming a second discrimination value for the second time span, 
when the second discrimination value is the same as the first 
discrimination value, then performing a new iteration with a new second 
time span that is formed by shortening the second time span of the 
preceding iteration, 

otherwise, ending the method and the trained neural network is the neural 
network of the last iteration wherein the second discrimination value is the 
same as the first discrimination value. 

2. A method according to claim 1, wherein an optimization method that is 
not gradient based is utilized for the maximization of at least one of the first 
discrimination value and of the second discrimination value. 


-3- 


3. A method according to claim 2, wherein the optimization method is 
based on the ALOPEX method. 

4. A method according to claim 1, whereby the first discrimination value 
I(T) satisfies the following rule: 



5 wherein 

• s references the input quantities, 

• t (n) references a pulse that is generated by a pulsed neuron n at a time m 

within a time span [0, T], 

• k„ (n = 1, N) references a point in time at which the pulsed neuron n has 
1 0 generated the last pulse within the time span [0, T], and 

• N references a plurality of pulsed neurons contained in the neural network. 

5. A method according to claim 4, wherein the first discrimination value 
I(T) satisfies the following rule: 


KJ) = -Jp(out) • ln(p( 0 nt))dt| 1) ...dtW...d4^ + 

♦ tpj p(c46)) . l^p(outp)>t«..dt«...dt<;) 

uuith 

p(oUt) =6 

wherein 

• references an input quantity that is applied to the neural network at a 
timej, 

• pj references a probability that the input quantity is applied to the 
neural network at a pint in time j, 

• p(out | -*) references a conditioned probability that a pulse is generated 
by a pulsed neuron in the neural network under the condition that the input 
quantity is applied to the neural network at a point in time j. 

6. A method according to claim 1, wherein the training sequence of inputs 
quantities are is of measured physical signals. 

7. A method according to claim 6, wherein the training sequence of input 
quantities is signals of an electroencephalogram. 


8. A method for classification of a sequence of input quantities upon 
employment of a neural network that contains pulsed neurons and was trained, 
comprising to the following steps: 


a) training the neural network for a first time span such that a discrimination 
value is maximized, as a result whereof a maximum first discrimination 
value is formed; 

b) forming the discrimination value dependent on pulses that are formed by 
the pulsed neurons within a first time span as well as on a training 
sequence of input quantities that are supplied to the neural network; 

c) implementing the following steps interactively: 
shortening the first time span to form a second time span, 
forming a second discrimination value for the second time span, 
when the second discrimination value is the same as the first 
discrimination value, then performing a new iteration with a new second 
time span that is formed by shortening the second time span of the 
preceding iteration, 

otherwise, ending the method and the trained neural network is the neural 
network of the last iteration wherein the second discrimination value is the 
same as the first discrimination value, 
supplying the sequence of input quantities to the neural network; and 
forming a classification signal that indicates what kind of sequence of input 
quantities the supplied sequence is. 

9. A method according to claim 9, wherein the training sequence of input 
quantities and the sequence of input quantities are measured physical signals. 

10. A method according to claim 9, wherein the training sequence of input 
quantities and the sequence of input quantities are measured signals of an 
electroencephalogram . 


1 1 . A neural network that contains pulsed neurons has been trained 
according to the following steps: 

a) the neural network is trained such for a first time span that a 
discrimination value is maximized, as a result whereof a maximum first 
discrimination value is formed; 

b) the discrimination value is formed dependent on pulses that are formed by 
the pulsed neurons within the first time span as well as on a training 
sequence of input quantities that are supplied to the neural network; 

c) the following steps are interactively implemented: 

the first time span is shortened to form a second time span, 

a second discrimination value is formed for the second time span, 

when the second discrimination value is the same as the first 

discrimination value, then a new iteration ensues with a new second time 

span that is formed by shortening the second time span of the preceding 

iteration, 

otherwise, the method is ended and the trained neural network is the neural 
network of the last iteration wherein the second discrimination value is the 
same as the first discrimination value. 

12. A neural network according to claim 10, wherein the network is 
utilized for classification of a physical signal. 

13. A neural network according to claim 10, utilized for the classification 
of an electroencephalogram signal. 

14. An arrangement for training a neural network that contains pulsed 
neurons comprising: 


a processor that is configured such that the following steps implemented: 

a) the neural network is trained such for a first time span that a 
discrimination value is maximized, as a result whereof a maximum first 
discrimination value is formed; 

b) the discrimination value is formed dependent on pulses that are formed by 
the pulsed neurons within the first time span as well as on a training 
sequence of input quantities that are supplied to the neural network; 

c) the following steps are interactively implemented: 

the first time span is shortened to form a second time span, 
a second discrimination value is formed for the second time span, 
when the second discrimination value is the same as the first 
discrimination value, then a new iteration ensues with a new second time 
span that is formed by shortening the second time span of the preceding 
iteration, and 

otherwise, the method is ended and the trained neural network is the neural 
network of the last iteration wherein the second discrimination value is the 
same as the first discrimination value. 

15. An arrangement according to claim 14, wherein the network is utilized 
for the classification of a physical signal. 

16. An arrangement according to claim 14, wherein the network is utilized 
for the classification of a signal of an electroencephalogram. 


REMARKS 


The foregoing amendments to the specification and claims under Article 
41 of the Patent Cooperation Treaty place the application into a form for 
prosecution before the U.S. Patent and Trademark Office under 35 U.S.C. §371. 
Accordingly, entry of these amendments before examination on the merits is 
hereby requested. 


I^efvin A. Robinson (reg. no. 31,870) 
Schiff Hardin & Waite 
Patent Department 
6600 Sears Tower 
Chicago, Illinois 60606 
Telephone: 312-258-5785 

ATTORNEY FOR APPLICANT 


Respectfully submitted, 
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SPECIFICATION 


TITLE 


METHOD FOR TRAINING A NEURAL NETWORK, METHOD FOR 
THE CLASSIFICATION OF A SEQUENCE OF INPUT QUANTITIES 
UPON EMPLOYMENT OF A NEURAL NETWORK, NEURAL 
NETWORK AND ARRANGEMENT FOR THE TRAINING 


Field of the Invention 

The present invention is directed to a method for training a neural network, to 
a method for the classification of a sequence of input quantities upon employment of a 
neural network as well as to a neural network and an arrangement for training a neural 
network. 

Description of the Related Art 

A neural network comprises neurons that are at least partially connected to one 
another. Input neurons of the neural network are supplied with input signals as input 
quantities supplied to the input neurons. The neural network usually comprises a 
plurality of layers. A respective neuron generates a signal dependent on input 
quantities supplied to a neuron of the neural network and on an activation function 
provided for the neuron, the signal being in turn supplied to neurons of a further layer 
as an input quantity according to a prescribable weighting. An output quantity 
dependent on quantities that are supplied to the output neuron of neurons of the 
preceding layer is generated in an output neuron in an output layer. There are 
currently essentially two approaches in view of the questions as to the form in which 
information is stored in a neural network. 

A first approach assumes that the information in a neural network is encoded 
in the spectral domain. Given this approach, a chronological sequence of input 
quantities is encoded such that a respective input neuron is provided for each time row 
value of a chronological sequence of the input quantities, the respective time row 
value being applied to this input neuron. 


OF A NEURAL NETWORK 


BACKGROUND OF THE INVENTION 


2 

Given a neural network that is designed according to this approach, a 
hyperbolic tangent (tanh) is usually employed as an activation function. 

This first type of neural network is referred to below as a static neural network. 

What is particularly disadvantageous about this approach is that it is not 
5 possible with a static neural network to explicitly consider a dynamics of a process 
subject to a technical system in the internal coding of the sequence of input quantities. 

The Time Delay Neural Networks (TDNN) known from German Patent 
Document DE 195 31 697 C2 attempt to counter this disadvantage in that, given a 
plurality of sequences of input quantities, a respective input neuron is provided for 
1 0 each sequence and for each time row value. This approach particularly exhibits the 
disadvantage that the dimension of the input space ~ represented by the plurality of 
input neurons — increases exponentially with an increasing plurality of different 
sequences of input quantities to be taken into consideration. 

An increasing plurality of neurons in the neural network, moreover, involves 
15 an increased training outlay upon employment of a plurality of training data that 

increases with an increasing plurality of neurons. A training of a static neural network 
becomes highly calculation-intensive under these conditions or, respectively, can 
practically no longer be implemented. 

A gradient-based training method, for example the back-propagation method, 
2 0 is usually utilized for training a static neural network. 

The publication by Unnikrishnan, et al., "Alopex: A Correlation-Based 
Learning Algorithm for Feedforward and Recurrent Neural Networks", also discloses 
a training method for a static neural network that is referred to as the ALOPEX 
method. In this method, the training of a static neural network is viewed as an 
2 5 optimization problem. In this case, the goal of the optimization is a minimization of a 
error criterion E taking weightings that are present in the static neural network and 
with which the connections between neurons are weighted into consideration for a 
predetermined training data set with training data. 


3 


A training datum is a tuple that has input quantities, for example state 
quantities of a technical system or, respectively, boundary conditions that a technical 
system is subject to and that are supplied to a technical system as well as an output 
quantity determined under the boundary conditions and that the technical system 
5 forms for the input quantities. 

The ALOPEX method shall be explained in greater detail later in conjunction 
with the exemplary embodiment. 

A second approach can be seen therein that the information about a system is 
encoded in the time domain and in the spectral domain. An artificial neural network 
1 0 that does justice to this approach comprises what are referred to as pulsed neurons and 
is known from the publication by Gerstner, "Time Structure of the Activity in Neural 
Networks Models". 

According to the publication by Deco et al., "Information Transmission and 
Temporal Code in Central Spiking Neurons", a pulsed neuron is modelled such that 
1 5 the behavior of a pulsed neuron with respect to an external stimulation, which is 
referred to below as an input quantity, is described by a stochastic differential 
equation of the Ito type according to the following rule: 


In the rule (1), dW(t) references a standard Wiener process. A predetermined 
constant x describes a delay of a membrane potential V(t) of the modelled neuron 
2 0 without an input quantity that is adjacent at the neuron. The model simulates the 

behavior of a biological neuron. For this reason, a pulsed neuron is also referred to as 
a biologically oriented neuron. 

Further, S(t) references a coupling of the neuron with another neuron, i.e. the 
following applies: 



(1) 


a(t) = |;S(t) = S^t-ti), 


i 


whereby t ; references an arrival time at which an external impulse arrives at an input 
of a neuron. A soma-synaptic intensity is modelled by a synaptic quantity w. 

In this model, the pulsed neuron generates a pulse when the membrane 
potential V(t) reaches a predetermined threshold 0. After the pulse is generated, the 
membrane potential V(t) of the neuron is reset to a predetermined initialization 
potential value V(0). 

A time sequence of pulses is thus described according to the following rule: 


It is also known from the afore-mentioned Deco et al. publication that, given 
the assumption of the above-described model for a pulsed neuron, a discrimination 
value I(T) can be formed that indicates the dependability with which a sequence of 
input quantities is correctly classified in view of the training data employed for a 
training of the neural network. 

The discrimination value I(T) is dependent on pulses that are formed by the 
pulsed neurons within a time span [0; T] as well as on a training sequence of input 
quantities that are supplied to the neural network. The discrimination value I(T) 
satisfies the following rule: 


to— /tk 


and satisfies the following rule: 



l(T) = I s; 



whereby 


s references the input quantities, 


5 

• references a pulse that is generated by a pulsed neuron n at a time m 

within a time span [0, T], 

• kn (n = 1, N) references a point in time at which the pulsed neuron n has 
generated the last pulse within the time span [0, T], 

• N references a plurality of pulsed neurons contained in the neural network. 
A stochastic differential equation of the Ito type derives for a neural network 

with a plurality of N neurons described according to the following rule: 

dVj.(t) = f- + jtjdt + odWi(t) + 


+ iw ij z8[t-t^_ Ai .)dt + i i (t)dt / 


whereby 

• Vj (t) references a membrane potential of the i & neuron (i = 1, N), 

• N references a plurality of neurons contained in the neural network, 

• Wjj respectively references a weighting of a coupling between the i th and the j* 
neuron, clearly a synaptic intensity between the neurons i and j, 

• Ay references a prescribable axional delay of a signal between the neurons i 
and j, 

• I,(t) references an external stimulation signal of the neuron i. 

The German Patent Document DE 195 31 967C2 discloses a training method 
for a neural network. Given this method, the neural network is linked in a control 
circuit with the model of a technical system such that the neural network outputs at 
least one manipulated variable to the model as an output quantity, and the model 
generates at least one regulating variable from the manipulated quantity supplied by 
the neural network, the at least one regulating variable being supplied to the neural 
network as an input quantity. The regulating variable is superimposed with a noise 
having a known noise distribution before it is supplied to the model. As a reaction to 
the regulating variable modified by the impressed noise, the weightings of the neural 


network are set as follows: A cost function evaluates whether the change in weighting 
at the network has effected an improvement of the regulating variable with respect to 
a rated behavior of the model, and such weightings are favored by the cost function. 

SUMMARY OF THE INVENTION 

The present invention is based on providing a method as well as an 
arrangement for training a neural network having pulsed neurons. The invention is 
also based on providing a method for the classification of a sequence of input 
quantities upon employment of a neural network having pulsed neurons as well as 
specifying a neural network having pulsed neurons. 

The problems are solved by the methods and the arrangement as well as by the 
neural network trained according to a method for training a neural network that 
contains pulsed neurons, 

a) the neural network is trained such for a first time span [0; T] that a 
discrimination value is maximized, as a result whereof a maximum first 
discrimination value is formed; 

b) the discrimination value is formed dependent on pulses that are formed by the 
pulsed neurons within the first time span as well as on a training sequence of 
input quantities that are supplied to the neural network; 

c) the following steps are interactively implemented: 

- the first time span is shortened to form a second time span, 

- a second discrimination value is formed for the second time span, 

- when the second discrimination value is the same as the first discrimination 
value, then a new iteration ensues with a new second time span that is formed 
by shortening the second time span of the preceding iteration, 

- otherwise, the method is ended and the trained neural network is the neural 
network of the last iteration wherein the second discrimination value is the 
same as the first discrimination value. 


In the preferred method, an optimization method that is not gradient based is 
utilized for the maximization of the first discrimination value and/or of the second 
discrimination value. The optimization method may be based on the ALOPEX 
method. 

According to one aspect of the invention, the first discrimination value I(T) 
satisfies the following rule: 



whereby 

• s references the input quantities, 

• *X n ) references a pulse that is generated by a pulsed neuron n at a time m 

within a time span [0, T], 

• k n (n= 1, N) references a point in time at which the pulsed neuron n has 
generated the last pulse within the time span [0, T], and 

• N references a plurality of pulsed neurons contained in the neural network. 
The first discrimination value I(T) in a preferred embodiment satisfies the 
following rale: 

I(T) = ~Jp(out) • In^out^tP... dtgj>... dtg + 


with 


p(cmt) = X P jp[outW} , 
j = l 


whereby 


• s^ J -* references an input quantity that is applied to the neural network at a time 
j, 

• pj references a probability that the input quantity ^ ^ is applied to the neural 
network at a pint in time j, 

• p(out | s*- J ^) references a conditioned probability that a pulse is generated by a 
pulsed neuron in the neural network under the condition that the input quantity 
s^ J -* is applied to the neural network at a point in time j . 

According to the foregoing, the training sequence of inputs quantities is 
measured physical signals, in particular, the training sequence of input quantities is 
signals of an electroencephalogram. 

The present invention provides a method for the classification of a sequence of 
input quantities upon employment of a neural network that contains pulsed neurons 
and was trained according to the following steps: 

a) the neural network is trained such for a first time span that a discrimination 
value is maximized, as a result whereof a maximum first discrimination value 
is formed; 

b) the discrimination value is formed dependent on pulses that are formed by the 
pulsed neurons within the first time span as well as on a training sequence of 
input quantities that are supplied to the neural network; 

c) the following steps are interactively implemented: 

- the first time span is shortened to form a second time span, 

- a second discrimination value is formed for the second time span, 

- when the second discrimination value is the same as the first discrimination 
value, then a new iteration ensues with a new second time span that is formed 
by shortening the second time span of the preceding iteration, 

- otherwise, the method is ended and the trained neural network is the neural 
network of the last iteration wherein the second discrimination value is the 
same as the first discrimination value, 

— whereby the sequence of input quantities is supplied to the neural network; 


- whereby a classification signal is formed that indicates what kind of sequence of 
input quantities the supplied sequence is. 

In the preferred embodiment, the training sequence of input quantities and the 
sequence of input quantities are measured physical signals, for example, measured 
signals of an electroencephalogram. 

The present invention also provides a neural network that contains pulsed 
neurons has been trained according to the following steps: 

a) the neural network is trained such for a first time span that a discrimination 
value is maximized, as a result whereof a maximum first discrimination value 
is formed; 

b) the discrimination value is formed dependent on pulses that are formed by the 
pulsed neurons within the first time span as well as on a training sequence of 
input quantities that are supplied to the neural network; 

c) the following steps are interactively implemented: 

- the first time span is shortened to form a second time span, 

- a second discrimination value is formed for the second time span, 

- when the second discrimination value is the same as the first discrimination 
value, then a new iteration ensues with a new second time span that is formed 
by shortening the second time span of the preceding iteration, 

- otherwise, the method is ended and the trained neural network is the neural 
network of the last iteration wherein the second discrimination value is the 
same as the first discrimination value. 

The neural network is utilized for the classification of a physical signal, such 
as classification of a signals of an electroencephalogram. 

An arrangement for training a neural network that contains pulsed neurons 
comprises a processor that is configured such that the following steps can be 
implemented: 


a) the neural network is trained such for a first time span that a discrimination 
value is maximized, as a result whereof a maximum first discrimination value 
is formed; 

b) the discrimination value is formed dependent on pulses that are formed by the 
pulsed neurons within the first time span as well as on a training sequence of 
input quantities that are supplied to the neural network; 

c) the following steps are interactively implemented: 

- the first time span is shortened to form a second time span, 

- a second discrimination value is formed for the second time span, 

- when the second discrimination value is the same as the first discrimination 
value, then a new iteration ensues with a new second time span that is formed 
by shortening the second time span of the preceding iteration, 

- otherwise, the method is ended and the trained neural network is the neural 
network of the last iteration wherein the second discrimination value is the 
same as the first discrimination value. 

The arrangement may be utilized for the classification of a physical signal, 
such as for the classification of a signal of an electroencephalogram. 

A method for training a neural network that contains pulsed neurons comprises 
the following steps: 

a) the neural network is trained for a first time span such that a discrimination 
value is maximized, as a result whereof a maximum first discrimination value 
is formed; 

b) the discrimination value is formed dependent on pulses that are formed by the 
pulsed neurons within the first time span as well as on a training sequence of 
input quantities that are supplied to the neural network; 

c) the following steps are interactively implemented: 

- the first time span is shortened to form a second time span, 

- a second discrimination value is formed for the second time span, 


— when the second discrimination value is the same as the first discrimination 
value, then a new iteration ensues with a new second time span that is formed 
by shortening the second time span of the preceding iteration, 

— otherwise, the method is ended and the trained neural network is the neural 
network of the last iteration wherein the second discrimination value is the 
same as the first discrimination value. 

A method for the classification of a sequence of input quantities upon 
employment of a neural network that contains pulsed neurons and was trained 
according to the following steps comprises the following steps: 

a) the neural network is trained for a first time span such that a discrimination 
value is maximized, as a result whereof a maximum first discrimination value 
is formed; 

b) the discrimination value is formed dependent on pulses that are formed by the 
pulsed neurons within the first time span as well as on a training sequence of 
input quantities that are supplied to the neural network; 

c) the following steps are interactively implemented: 

— the first time span is shortened to form a second time span, 

~ a second discrimination value is formed for the second time span, 

— when the second discrimination value is the same as the first discrimination 
value, then a new iteration ensues with a new second time span that is formed 
by shortening the second time span of the preceding iteration, 

— otherwise, the method is ended and the trained neural network is the neural 
network of the last iteration wherein the second discrimination value is the 
same as the first discrimination value, 

— the sequence of input quantities is supplied to the neural network; 

d) a classification signal is formed that indicates what kind of sequence of input 
quantities the supplied sequence is. 

A neural network that contains pulsed neurons has been trained according to 
the following steps: 


a) the neural network is trained for a first time span such that a discrimination 
value is maximized, as a result whereof a maximum first discrimination value 
is formed; 

b) the discrimination value is formed dependent on pulses that are formed by the 
pulsed neurons within the first time span as well as on a training sequence of 
input quantities that are supplied to the neural network; 

c) the following steps are interactively implemented: 

— the first time span is shortened to form a second time span, 

— a second discrimination value is formed for the second time span, 

— when the second discrimination value is the same as the first discrimination 
value, then a new iteration ensues with a new second time span that is formed 
by shortening the second time span of the preceding iteration, 

— otherwise, the method is ended and the trained neural network is the neural 
network of the last iteration wherein the second discrimination value is the 
same as the first discrimination value. 

An arrangement for training a neural network that contains pulsed neurons 
comprises a processor that is configured such that the following steps can be 
implemented: 

a) the neural network is trained for a first time span such that a discrimination 
value is maximized, as a result whereof a maximum first discrimination value 
is formed; 

b) the discrimination value is formed dependent on pulses that are formed by the 
pulsed neurons within the first time span as well as on a training sequence of 
input quantities that are supplied to the neural network; 

c) the following steps are interactively implemented: 

— the first time span is shortened to form a second time span, 

— a second discrimination value is formed for the second time span, 


— when the second discrimination value is the same as the first discrimination 
value, then a new iteration ensues with a new second time span that is formed 
by shortening the second time span of the preceding iteration, 

— otherwise, the method is ended and the trained neural network is the neural 
network of the last iteration wherein the second discrimination value is the 
same as the first discrimination value. 

The invention makes it possible to classify a time sequence of input quantities 
with a neural network what contains pulsed neurons, whereby it is assured that, given 
optimized classification dependability, a minimized plurality of time values must be 
supplied to the neural network for classification. 

Preferred developments of the invention INSERT DEPENDENT CLAIMS. 

An optimization method that is not gradient based is preferably employed for 
the maximization of the first discrimination value and/or the second discrimination 
value, preferably an optimization method based on the ALOPEX method. 

The first discrimination value preferably satisfies the following rule: 



whereby 

• s references the input quantities, 

• t jn) references a pulse that is generated by a pulsed neuron n at a time m 

within a time span [0, T], 

• k n (n = 1, N) references a point in time at which the pulsed neuron n has 
generated the last pulse within the time span [0, T], 

• N references a plurality of pulsed neurons contained in the neural network. 
In a further development, the first discrimination value satisfies the following 


14 


1(1!) = -J p(out) • li^outfldtj 0 ... dtg. .. dtjg + 

+ i Pj j P (o4e)) . Jp(outW)1dtp... dtg... 


with 

p(out) = £pjp 


whereby 

• J -* references an input quantity that is applied to the neural network at a time 
j, 

• pj references a probability that the input quantity s^ is applied to the neural 
network at a pint in time j, 

• p(out | s^ J ^) references a conditioned probability that a pulse is generated by a 
pulsed neuron in the neural network under the condition that the input quantity 

is applied to the neural network at a point in time j. 
The training sequences of input quantities are preferably measured physical 

signals. 

The methods and the arrangements can thus be utilized in the framework of the 
description of a technical system, particularly for describing or, respectively, 
investigating a multi-channel signal that has been registered by an 
electroencephalograph and describes an electroencephalogram. 

The methods and the arrangements can also be utilized for the analysis of 
multi-variant financial data in a financial market for the analysis of economic 
relationships. 

The described method steps can be realized both in software for the processor 
as well as in hardware, i.e. with a specific circuit. 
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BRIEF DESCRIPTION OF DRAWINGS 

An exemplary embodiment of the invention is shown in the Figures and is 
explained in greater detail below. 

Figure 1 is a flowchart wherein the individual method steps of the exemplary 
5 embodiment are presented; 

Figure 2 is a functional block diagram showing an electroencephalograph and a 
patient for whom a electroencephalogram is produced; 

Figure 3 is a schematic diagram of a neural network according to the exemplary 
embodiment; 

1 0 Figure 4 is a schematic diagram on the basis whereof the principle underlying the 
exemplary embodiment is shown. 

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 
Figure 2 shows a patient 200 to whose head 201 sensors 202, 203, 204, 205 
and 206 are attached for the registration of brain currents. Electrical signals 207, 208, 

15 209, 210 and 21 1 picked up by the sensors 202, 203, 204, 205, 206 are supplied to an 
electroencephalograph 220 via a first input/output interface 221. The 
electroencephalograph 220 comprises a plurality of input channels. Via the 
input/output interface 221, which is connected to an analog-to-digital converter 222, 
the electrical signals are supplied to the electroencephalograph 220 and digitalized in 

2 0 the analog-to-digital converter 222, and each registered electrical signal is stored in a 
memory 223 as a sequence of time row values. 

A sequence of time row values is thus characterized by a sampling interval as 
well as by a time duration, referred to below as a time span, during which a respective 
electrical signal is registered. The memory 223 is connected to the analog-to-digital 

2 5 converter 222 as well as to a processor 224 and a second input-output interface 225 
via a bus 226. 
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A picture screen 228 (via a first cable 227), a keyboard 230 (via a second cable 
229) and a computer mouse 232 (via a third cable) are also connected to the second 
input/output interface 225. 

Results of the examination of the patient 200 are shown on the picture screen 
5 228. A user (not shown) can make inputs into the system via the keyboard 230 or, 
respectively, the computer moose 232. 

The processor 224 is configured such that the method steps described later can 
be implemented. 

A respective sequence of time row values as well as particular information 
1 0 about the class of time row values to which the sequence of time row values is to be 
allocated form a training datum. 

A plurality of training data form a training data set with which a neural 
network 301 described later is trained. 

Fi gure 3 shows the neural network 301 with pulsed neurons. 
15 A sequence 306, 307 and 308 of time row values is respectively applied to a 

respective input neuron 302, 303 and 304 of an input layer 305. A particular 
information as to whether the sequence 306, 307 and 308 of the time row values, 
broadly referred to as input pattern 306, 307 and 308, is a matter of an input pattern 
306, 307 and 308 of a first class or a matter of an input pattern 306, 307 and 308 of a 
2 0 second class is allocated to each applied sequence 306, 307 and 308 of time row 
values in the framework of the training method. 

A respective input neuron 302, 303 and 304 is respectively connected to an 
intermediate neuron 309, 310 and 31 1 of an intermediate layer 312 via a weighted 
connection 313, 314 and 315. 
2 5 The intermediate neurons 309, 310 and 3 1 1 are connected to one another via 

connections 316, 317, 318, 319, 320 and 321 that are likewise weighted. 

The intermediate neurons 309, 310 and 311 are also connected to further 
pulsed neurons 322, 323 and 324 via weighted connections 325, 326, 327, 328, 329 
and 330. 
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The pulsed neurons respectively comprise the above-described behavior that is 
presented in the publication by Gerstner, "Time Structure of the Activity in Neural 
Network Models". 

The intermediate neurons 309, 3 1 0 and 311 are connected to a plurality of the 
5 intermediate neurons 309, 3 1 0 and 3 1 1 ; the respective, further pulsed neurons 322, 

323 and 324 are respectively connected to exactly one intermediate neuron 309, 310, 
311. In this way, it is possible to model a far-reaching influencing between neurons 
of a neural network as well as a local influencing of neurons within the neural 
network. 

1 0 An output neuron 331 is connected to the further pulsed neurons 322, 323 and 

324 via weighted connections 332, 333 and 334. The output neuron 331 forms an 
output signal 335 that indicates the class to which the input pattern 306, 307 and 308 
belongs. 

In the training phase of the neural network 301, the output quantity 335 is 
1 5 compared to the classification particular information allocated to the respective input 
pattern, and an error signal E is formed that is employed for adapting the weightings 
of the connections between the neurons present in the neural network 301 . 

The method according to the ALOPEX method, which is not gradient based, is 
utilized as the training method in the framework of this exemplary embodiment. The 
2 0 goal of the ALOPEX method is the minimization of an error criterion E taking into 
consideration and adapting the weightings w tc for a training dataset. 
The ALOPEX is explained in greater detail below. 

A neuron b is connected to a neuron c via a connection that is weighted with 
the weighting w bc . During an f* iteration, the weighting w bc is updated according to 
2 5 the following rule: 

Wbc(f) = «bc(* - 1) + *bc(f) f (10) 


whereby 5 bc (f) references a small positive or negative, predetermined step width 8 
according to the following rule: 
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M = \~ 5 wftho P robabflit Y p bc (f) 
bc ^ tulth q probability 1 ~ Pbc(f) ' 


A probability p bc (f) is formed according to the following rule: 

Pbc(f) = 15^' fl2 > 


1 + e 


T{f) 


whereby C bc (f) is formed according to the following rule: 

C bc (f) = Aw bc (f) • AE(f). (13) 

T(f) references a prescribable value. Aw bc (f) and AE(f) reference the weighting 
changes Aw k (f) of the weightings w bc or, respectively, the change AE(f) of the error 
criterion during the preceding two iterations according to the rules: 

Aw bc (f) = w bc (f - 1} + w bc (f - 2), (14) 

AE bc (f) = E bc (f - 1) + E bc (£ - 2) . (15) 

The predetermined value T(f) is updated every F iterations according to the following 
rule: 

w - £ z z sWi as) 

be f*=t-F 
when f is a whole multiple of F, and 
T(f) = T(f - 1) othenuise, (17) 

whereby M references a plurality of connections in the neural network 301. 
Equation (16) can be simplified to form the following rule: 


The neural network 301 is trained according to the above-described training 
method upon employment of the training dataset. 

Further, a first discrimination value I(T) for the neural network 301 is formed 
according to the following rule: 



(19) 


whereby 

• s references the input quantities, 

• t (^) references a pulse that is generated by a pulsed neuron n at a time m 

within a time span [0, T], 

• k n (n = 1, N) references a point in time at which the pulsed neuron n has 
generated the last pulse within the time span [0, T], 

• N references a plurality of pulsed neurons contained in the neural network. 
The first discrimination value I(T) clearly corresponds to the difference of the 

following entropies: 
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I(T) = H(out)- {H(o^t|s)) s , (20) 
mith 

H(out) = -J p(out) •: lnjpfoutjjdt^... dt^... dt^ {21 ) 

and 

(HCoutts)),, = -£ Pi j p[out|s6)) • ln(p(ootp))dtP... dtg... *tg . 

(22) 


The first discrimination value I(T) thus derives according to the following 

rule: 
whereby 

• s^ J references an input quantity that is applied to the neural network at a time 
5 j, 

• Pj references a probability that the input quantity ^ is applied to the neural 
network at a pint in time j, 

• p(out| s^ references a conditioned probability that a pulse is generated by a 
pulsed neuron in the neural network under the condition that the input quantity 

10 s*- ^ is applied to the neural network at a point in time j . 

When a maximum first discrimination value I(T) has been determined in the 

framework of training the neural network 301, then this means that the input pattern 

306, 307, 308 observed in the first time span contains enough information in order to 

classify the input pattern with adequate dependability. 
15 The first discrimination value I(T) is clearly formed (step 101) in the 

framework of the training for a first time span [0; T] (see Fi gure 1 ). 
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In a further step (step 102), a second time span is formed by shortening the 
first time span: [0; T], whereby T < T applies. 

For the second time span [0; T], a second discrimination value I(T') is formed 
in a further step (step 103) in the same way as described above for the first 
5 discrimination value I(T). 

The first discrimination value I(T) is compared to the second discrimination 
value I(T') (step 104). 

When the second discrimination value I(T') is the same as the first 
discrimination value I(T), then a new second time span is formed (step 105) by 
1 0 shortening the second time span [0; T'], and the new second time span is considered 
to be the second time span (step 106). A second discrimination value I(T') is in turn 
formed (step 103) for the second time span of the new iteration. 

Clearly, this iterative method means that the time span wherein pulses 
generated by the pulsed neurons are taken into consideration for forming the output 
1 5 signal is shortened until the second discrimination value I(T') is unequal to the first 
discrimination value I(T). 

When the second discrimination value I(T') is smaller than the first 
discrimination value, then the neural network 301 is viewed as being an optimized 
neural network that was trained in the last preceding iteration wherein the second 
2 0 discrimination value I(T') was not smaller than the first discrimination value I(T) 
(step 107). 

The time span respectively taken into consideration is divided into discrete 
time sub-spans for which the only thing respectively determined is whether a neuron 
generated a pulse during the time sub-span or not. 
2 5 In this way, the calculating outlay needed for the training is considerably 

reduced. 

For further illustration, the principle is explained again on the basis of Figure 

4. 
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Fi gure 4 shows two continuous processes pi and p2 that are formed by a set 
of continuous input signals SI and S2. Two sequences of input quantities, the input 
patterns, are present after a corresponding, above-described digitalization. The input 
patterns are supplied to the trained neural network 401 in an application phase, and a 
5 space-time encoding of the processes pi and p2 is clearly implemented on the basis of 
the time rows for the trained neural network 401 . 

On the basis of an output signal 402, the trained neural network 401 indicates 
the kind of process the input pattern involves. The trained neural network 401 
exhibits the property that, first, the dependability of the optimization is optimized and, 
1 0 second, a minimum plurality of time row values, i.e. a minimum second time span 
403, is required in order to dependably implement the classification. 

A few alternatives to the above-described exemplary embodiment are present 

below: 

The plurality of inputs, of pulsed neurons as well as output signals is generally 
15 arbitrary. The plurality of different sequences of time row values is also arbitrary in 
the framework of the classification and in the framework of the training. An 
electroencephalogram analysis is thus possible for an arbitrary plurality of channels 
for characterizing tumors. 

Although other modifications and changes may be suggested by those skilled 
2 0 in the art, it is the intention of the inventors to embody within the patent warranted 
hereon all changes and modifications as reasonably and properly come within the 
scope of their contribution to the art. 


Abstract of the Disclosure 

For a first time span, the neural network is trained such that a discrimination 
value is maximized, whereby the discrimination values is dependent on pulses that are 
formed by pulsed neurons within the first time span. Iteratively, the first time span is 
shortened and a second discrimination value is formed until the second discrimination 
value is smaller than the maximum discrimination value. The trained neural network 
is the neural network of the last iteration wherein the second discrimination value is 
equal to the maximum discrimination value. 
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METHOD FOR TRAINING A NEURAL NETWORK, METHOD FOR THE 
CLASSIFICATION OF A SEQUENCE OF INPUT QUANTITIES UPON 
EMPLOYMENT OF A NEURAL NETWORK, NEURAL NETWORK AND 
ARRANGEMENT FOR THE TRAINING OF A NEURAL NETWORK 

The invention is directed to a method for training a neural network, to a 
method for the classification of a sequence of input quantities upon employment of a 
neural network as well as to a neural network and an arrangement for training a neural 
network. 

A neural network comprises neurons that are at least partially connected to 
one another. Input neurons of the neural network are supplied with input signals as 
input quantities supplied to the input neurons. The neural network usually comprises 
a plurality of layers. A respective neuron generates a signal dependent on input 
quantities supplied to a neuron of the neural network and on an activation function 
provided for the neuron, said signal being in turn supplied to neurons of a further 
layer as input quantity according to a prescribable weighting. An output quantity 
dependent on quantities that are supplied to the output neuron of neurons of the 
preceding layer is generated in an output neuron in an output layer. There are 
currently essentially two approaches in view of the questions as to the form in which 
information is stored in a neural network. 

A first approach assumes that the information in a neural network is 
encoded in the spectral domain. Given this approach, a chronological sequence of 
input quantities is encoded such that a respective input neuron is provided for each 
time row value of a chronological sequence of the input quantities, the respective time 
row value being applied to this input neuron. 

Given a neural network that is designed according to this approach, a 
hyperbolic tangent (tanh) is usually employed as activation function. 

This first type of neural network is referred to below as static neural 

network. 
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What is particularly disadvantageous about this approach is that it is not 
possible with a static neural network to explicitly consider a dynamics of a process 
subject to a technical system in the internal coding of the sequence of input quantities. 

The Time Delay Neural Networks (TDNN) known from [4] attempt to 
5 counter this disadvantage in that, given a plurality of sequences of input quantities, a 
respective input neuron is provided for each sequence and for each time row value. 
This approach particularly exhibits the disadvantage that the dimension of the input 
space - represented by the plurality of input neurons ~ increases exponentially with 
an increasing plurality of different sequences of input quantities to be taken into 
1 0 consideration. 

An increasing plurality of neurons in the neural network, moreover, 
involves an increased training outlay upon employment of a plurality of training data 
that increases with an increasing plurality of neurons. A training of a static neural 
network becomes highly calculation- intensive under these conditions or, respectively, 
1 5 can practically no longer be implemented. 

A gradient-based training method, for example the back-propagation 
method, is usually utilized for training a static neural network. 

[3] also discloses a training method for a static neural network that is 
referred to as the ALOPEX method. In this method, the training of a static neural 
2 0 network is viewed as an optimization problem. In this case, the goal of the 

optimization is a minimization of a error criterion E taking weightings that are present 
in the static neural network and with which the connections between neurons are 
weighted into consideration for a predetermined training data set with training data. 

A training datum is a tuple that [...] input quantities, for example state 

2 5 quantities of a technical system or, respectively, boundary conditions that a technical 

system is subject to and that are supplied to a technical system as well as an output 
quantity determined under the boundary conditions and that the technical system 
forms for the input quantities. 

The ALOPEX method shall be explained in greater detail later in 

3 0 conjunction with the exemplary embodiment. 


A second approach can be seen therein that the information about a system 
is encoded in the time domain and in the spectral domain. An artificial neural 
network that does justice to this approach comprises what are referred to as pulsed 
neurons and is known from [2]. 

According to [1 ], a pulsed neuron is modelled such that the behavior of a 
pulsed neuron with respect to an external stimulation, which is referred to below as 
input quantity, is described by a stochastic differential equation of the Ito type 
according to the following rule: 

dV(t) = [~ ^ + vjdt + odW(t) + wdS(t) . (1) 

In the rule (1), dW(t) references a standard Wiener process. A 
predetermined constant x describes a delay of a membrane potential V(t) of the 
modelled neuron without input quantity that is adjacent at the neuron. The model 
simulates the behavior of a biological neuron. For this reason, a pulsed neuron is also 
referred to as biologically oriented neuron. 

Further, S(t) references a coupling of the neuron with another neuron, i.e. 
the following applies: 

s(t) = ^S(t) = S^t-ti), (2) 

whereby t; references an arrival time at which an external impulse arrives at an input 
of a neuron. A soma-synaptic intensity is modelled by a synaptic quantity w. 

In this model, the pulsed neuron generates a pulse when the membrane 
potential V(t) reaches a predetermined threshold 0. After the pulse is generated, the 
membrane potential V(t) of the neuron is reset to a predetermined initialization 
potential value V(0). 

A time sequence of pulses is thus described according to the following 

rule: 


to, tJc, 


(3) 
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and satisfies the following rule: 

o(t) = E^t-tJc). «> 

k 

It is also known from [1] that, given the assumption of the above- 
described model for a pulsed neuron, a discrimination value I(T) can be formed that 
indicates the dependability with which a sequence of input quantities is correctly 
5 classified in view of the training data employed for a training of the neural network. 

The discrimination value I(T) is dependent on pulses that are formed by 
the pulsed neurons within a time span [0; T] as well as on a training sequence of input 
quantities that are supplied to the neural network. The discrimination value I(T) 
satisfies the following rule: 



1 0 whereby 

s references the input quantities, 

t (p) references a pulse that is generated by a pulsed neuron n at a time m 
within a time span [0, T], 

k„ (n = 1, N) references a point in time at which the pulsed neuron n 
1 5 has generated the last pulse within the time span [0, T], 

N references a plurality of pulsed neurons contained in the neural network. 
A stochastic differential equation of the ltd type derives for a neural 
network with a plurality of N neurons described according to the following rule: 



+ jxidt + crdWi(t) + 


+ 2 «i j2 sft - 4 j) _ A Idt + ii(t)dt 

j=l k v J/ 


whereby 

Vj (t) references a membrane potential of the i th neuron (i = 1 , . . ., N), 
N references a plurality of neurons contained in the neural network, 
w r respectively references a weighting of a coupling between the i th and 
the j th neuron, clearly a synaptic intensity between the neurons i and j, 
Ai- references a prescribable axional delay of a signal between the neurons 
i andj, 

I ; (t) references an external stimulation signal of the neuron i. 

[4] discloses a training method for a neural network. Given this method, 
the neural network is linked such in a control circuit with the model of a technical 
system that the neural network outputs at least one manipulated variable to the model 
as output quantity, and the model generates at least one regulating variable from the 
manipulated quantity supplied by the neural network, said at least one regulating 
variable being supplied to the neural network as input quantity. The regulating 
variable is superimposed with a noise having a known noise distribution before it is 
supplied to the model. As a reaction to the regulating variable modified by the 
impressed noise, the weightings of the neural network are set as follows: A cost 
function evaluates whether the change in weighting at the network has effected an 
improvement of the regulating variable with respect to a rated behavior of the model, 
and such weightings are favored by the cost function. 

The invention is based on the problem of specifying a method as well as 
an arrangement for training a neural network having pulsed neurons. The invention is 
also based on the problem of specifying a method for the classification of a sequence 
of input quantities upon employment of a neural network having pulsed neurons as 
well as specifying a neural network having pulsed neurons. 

The problems are solved by the methods and the arrangement as well as 
by the neural network having the features of the independent patent claims. 

A method for training a neural network that contains pulsed neurons 
comprises the following steps: 


a) the neural network is trained such for a first time span that a 
discrimination value is maximized, as a result whereof a maximum first 
discrimination value is formed; 

b) the discrimination value is formed dependent on pulses that are formed by 
the pulsed neurons within the first time span as well as on a training 
sequence of input quantities that are supplied to the neural network; 

c) the following steps are interactively implemented: 

~ the first time span is shortened to form a second time span, 

— a second discrimination value is formed for the second time span, 

— when the second discrimination value is the same as the first 
discrimination value, then a new iteration ensues with a new second time 
span that is formed by shortening the second time span of the preceding 
iteration, 

— otherwise, the method is ended and the trained neural network is the 
neural network of the last iteration wherein the second discrimination 
value is the same as the first discrimination value. 

A method for the classification of a sequence of input quantities upon 
employment of a neural network that contains pulsed neurons and was trained 
according to the following steps comprises the following steps: 

a) the neural network is trained such for a first time span that a 
discrimination value is maximized, as a result whereof a maximum first 
discrimination value is formed; 

b) the discrimination value is formed dependent on pulses that are formed by 
the pulsed neurons within the first time span as well as on a training 
sequence of input quantities that are supplied to the neural network; 

c) the following steps are interactively implemented: 

— the first time span is shortened to form a second time span, 

— a second discrimination value is formed for the second time span, 
~ when the second discrimination value is the same as the first 
discrimination value, then a new iteration ensues with a new second time 


span that is formed by shortening the second time span of the preceding 
iteration, 

- otherwise, the method is ended and the trained neural network is the 
neural network of the last iteration wherein the second discrimination 
value is the same as the first discrimination value, 

- the sequence of input quantities is supplied to the neural network; 

d) a classification signal is formed that indicates what kind of sequence of 

input quantities the supplied sequence is. 

A neural network that contains pulsed neurons has been trained according 
to the following steps: 

a) the neural network is trained such for a first time span that a 
discrimination value is maximized, as a result whereof a maximum first 
discrimination value is formed; 

b) the discrimination value is formed dependent on pulses that are formed by 
the pulsed neurons within the first time span as well as on a training 
sequence of input quantities that are supplied to the neural network; 

c) the following steps are interactively implemented: 

-- the first time span is shortened to form a second time span, 

- a second discrimination value is formed for the second time span, 
~ when the second discrimination value is the same as the first 
discrimination value, then a new iteration ensues with a new second time 
span that is formed by shortening the second time span of the preceding 
iteration, 

- otherwise, the method is ended and the trained neural network is the 
neural network of the last iteration wherein the second discrimination 
value is the same as the first discrimination value. 

An arrangement for training a neural network that contains pulsed neurons 
comprises a processor that is configured such that the following steps can be 
implemented: 


a) the neural network is trained such for a first time span that a 
discrimination value is maximized, as a result whereof a maximum first 
discrimination value is formed; 

b) the discrimination value is formed dependent on pulses that are formed by 
the pulsed neurons within the first time span as well as on a training 
sequence of input quantities that are supplied to the neural network; 

c) the following steps are interactively implemented: 

— the first time span is shortened to form a second time span, 

~ a second discrimination value is formed for the second time span, 

— when the second discrimination value is the same as the first 
discrimination value, then a new iteration ensues with a new second time 
span that is formed by shortening the second time span of the preceding 
iteration, 

— otherwise, the method is ended and the trained neural network is the 
neural network of the last iteration wherein the second discrimination 
value is the same as the first discrimination value. 

The invention makes it possible to classify a time sequence of input 
quantities with a neural network what contains pulsed neurons, whereby it is assured 
that, given optimized classification dependability, a minimized plurality of time 
values must be supplied to the neural network for classification. 

Preferred developments of the invention derive from the dependent claims. 

An optimization method that is not gradient based is preferably employed 
for the maximization of the first discrimination value and/or the second discrimination 
value, preferably an optimization method based on the ALOPEX method. 

The first discrimination value preferably satisfies the following rule: 


{ 

I(T) = I ! 


,(1) t (l) t (l) t (2) f t (2) (2) 
An) t (n) t (n) (n) M <N) 


whereby 

• s references the input quantities, 
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+X n ) references a pulse that is generated by a pulsed neuron n at a time m 
L m 

within a time span [0, T], 

ka (n = 1 , . . ., N) references a point in time at which the pulsed neuron n 
has generated the last pulse within the time span [0, T], 
5 • N references a plurality of pulsed neurons contained in the neural network. 

In a further development, the first discrimination value satisfies the 
following rule: 

I(T) = -J p(out) - ln(p(out))dtP... dtg... dtjjjj + 

+ i Pj J P (ou t p) - ^t^)>tP...dt«...dt» ^ 

3 = 1 


uuith 


p(out) = 2-PjP 
whereby 


3=1 


s^ ) references an input quantity that is applied to the neural network at a 
time j, 

pj references a probability that the input quantity s ( j } is applied to the 
neural network at a pint in time j, 

p(out | s ( j references a conditioned probability that a pulse is generated 
by a pulsed neuron in the neural network under the condition that the input 
quantity s^ * is applied to the neural network at a point in time j. 
The training sequences of input quantities are preferably measured 
physical signals. 

The methods and the arrangements can thus be utilized in the framework 
of the description of a technical system, particularly for describing or, respectively, 
investigating a multi-channel signal that has been registered by an 
electroencephalograph and describes an electroencephalogram. 


10 

The methods and the arrangements can also be utilized for the analysis of 
multi-variant financial data in a financial market for the analysis of economic 
relationships. 

The described method steps can be realized both in software for the 
5 processor as well as in hardware, i.e. with a specific circuit. 

An exemplary embodiment of the invention is shown in the Figures and is 
explained in greater detail below. 
Shown are: 

Figure 1 a flowchart wherein the individual method steps of the exemplary 

1 o embodiment are presented; 

Figure 2 a sketch of an electroencephalograph and a patient for whom a 

electroencephalogram is produced; 
Figure 3 a sketch of a neural network according to the exemplary embodiment; 
Figure 4 a sketch on the basis whereof the principle underlying the exemplary 
1 5 embodiment is shown. 

Fi gure 2 shows a patient 200 to whose head 201 sensors 202, 203, 204, 
205 and 206 are attached for the registration of brain stomata [sic; should probably 
read "currents"]. Electrical signals 207, 208, 209, 210 and 21 1 picked up by the 
sensors 202, 203, 204, 205, 206 are supplied to an electroencephalograph 220 via a 

2 0 first input/output interface 221 . The electroencephalograph 220 comprises a plurality 

of input channels. Via the input/output interface 221, which is connected to an 
analog-to-digital converter 222, the electrical signals are supplied to the 
electroencephalograph 220 and digitalized in the analog-to -digital converter 222, and 
each registered electrical signal is stored in a memory 223 as a sequence or time row 

2 5 values. 

A sequence of time row values is thus characterized by a sampling interval 
as well as by a time duration, referred to below as time span, during which a 
respective electrical signal is registered. The memory 223 is connected to the analog- 
to-digital converter 222 as well as to a processor 224 and a second input-output 

3 0 interface 225 via a bus 226. 
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A picture screen 228 (via a first cable 227), a keyboard 230 (via a second 
cable 229) and a computer mouse 232 (via a third cable) are also connected to the 
second input/output interface 225. 

Results of the examination of the patient 200 are shown on the picture 
screen 228. A user (not shown) can make inputs into the system via the keyboard 230 
or, respectively, the computer moose 232. 

The processor 224 is configured such that the method steps described later 
can be implemented. 

A respective sequence of time row values as well as a particular about the 
class of time row values to which the sequence of time row values is to be allocated 
form a training datum. 

A plurality of training data form a training data set with which a neural 
network 301 described later is trained. 

Fi gure 3 shows the neural network 301 with pulsed neurons. 

A sequence 306, 307, 308 of time row values is respectively applied to a 
respective input neuron 302, 303, 304 of an input layer 305. A particular as to 
whether the sequence 306, 307, 308 of the time row values, broadly referred to as 
input pattern 306, 307, 308, is a matter of an input pattern 306, 307, 308 of a first 
class or a matter of an input pattern 306, 307, 308 of a second class is allocated to 
each applied sequence 306, 307, 308 of time row values in the framework of the 
training method. 

A respective input neuron 302, 303, 304 is respectively connected to an 
intermediate neuron 309, 310, 31 1 of an intermediate layer 312 via a weighted 
connection 313, 314, 315. 

The intermediate neurons 309, 310, 31 1 are connected to one another via 
connections 316, 317, 318, 319, 320, 321 that are likewise weighted. 

The intermediate neurons 309, 310, 31 1 are also connected to further 
pulsed neurons 322, 323, 324 via weighted connections 325, 326, 327, 328, 329 and 
330. 

The pulsed neurons respectively comprise the above-described behavior 
that is presented in [2]. 
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The intermediate neurons 309, 3 10, 31 1 are connected to a plurality of 
intermediate neurons 309, 310, 311; the respective, further pulsed neurons 322, 323, 
324 are respectively connected to exactly one intermediate neuron 309, 3 10, 3 1 1 . In 
this way, it is possible to model a far-reaching influencing between neurons of a 
5 neural network as well as a local influencing of neurons within the neural network. 

An output neuron 33 1 is connected to the further pulsed neurons 322, 323, 
324 via weighted connections 332, 333 and 334. The output neuron 331 forms an 
output signal 335 that indicates the class to which the input pattern 306, 307, 308 
belongs. 

10 In the training phase of the neural network 301, the output quantity 335 is 

compared to the classification particular allocated to the respective input pattern, and 
an error signal E is formed that is employed for adapting the weightings of the 
connections between the neurons present in the neural network 301 . 


1 5 based, is utilized as the training method in the framework of this exemplary 

embodiment. The goal of the ALOPEX method is the minimization of an error 
criterion E taking into consideration and adapting the weightings w bc for a training 
dataset. 


with the weighting w bc . During an I th iteration, the weighting w bc is updated according 
to the following rule: 


whereby 8 bc (f) references a small positive or negative, predetermined step width 5 
according to the following rule: 


The method according to the ALOPEX method, which is not gradient 


20 


The ALOPEX is explained in greater detail below. 

A neuron b is connected to a neuron c via a connection that is weighted 


wbc(f) = " bc (f " 1) + *bc(f), 


(10) 



uuith q probability 
uuith a probability 


Pbc(f) 

1 - Pbc(f) " 


(11) 
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A probability p bc (f) is formed according to the following rule: 


error 


whereby C bc (f) is formed according to the following rule: 

C bc (f) = Aw bc (f) • AE(f). (13) 

T(f) references a prescribable value. Aw bc (f) and AE(f) reference the weighting 
changes Aw bc (f) of the weightings w bc or, respectively, the change AE(f) of the - 
criterion during the preceding two iterations according to the rules: 

Aw bc (f) = w bc (f - 1) + w bc (f - 2), (14) 

AE bc (f) = E bc (f - 1) + E bc (f - 2) . (15) 

The predetermined value T(f) is updated every F iterations according to the following 
rule: 

m b C f« = f-F 
when f is a whole multiple of F, and 
T(f) = T(f - l) otherwise, (17) 

whereby M references a plurality of connections in the neural network 301 . 
Equation (16) can be simplified to form the following rule: 

f = f-F 

The neural network 301 is trained according to the above-described 
training method upon employment of the training dataset. 


Further, a first discrimination value I(T) for the neural network 301 is 
formed according to the following rule: 


I(T) 


( LCD t (i) t (D t (2) t (2), 

Jn) .(n) t (N) .(N) M\ 


whereby 

s references the input quantities, 

4-( n ) references a pulse that is generated by a pulsed neuron n at a time m 
'-in 

within a time span [0, T], 

k„ (n = 1 , N) references a point in time at which the pulsed neuron n 
has generated the last pulse within the time span [0, T], 
N references a plurality of pulsed neurons contained in the neural network. 
The first discrimination value I(T) clearly corresponds to the difference of 
the following entropies: 


I(T) = H(out) - <H(out|s)) s 


(20) 


H(out) = -Jp(out) • ln(p(out))dti l) ...dt^...d4 1 ^ 


and 


(H(out|s)) s = - £ Pj Jp(outp)) • l^out^ . 


j = l 


(22) 


The first discrimination value I(T) thus derives according to the following 

rule: 


I(T) = -J p(out) • ln(p(out))d4 l) ...d4 1 1 ) ...dt 


(N) 

k N 


j = l 


p(out) = £p jP (out|s<3)), (24) 
j = l 

whereby 

s ( j -* references an input quantity that is applied to the neural network at a 
time j, 

pj references a probability that the input quantity ^ is applied to the 
neural network at a pint in time j, 

p(out | s ( j references a conditioned probability that a pulse is generated 

by a pulsed neuron in the neural network under the condition that the input 

quantity s^ ^ is applied to the neural network at a point in time j. 

When a maximum first discrimination value I(T) has been determined in 
the framework of training the neural network 301, then this means that the input 
pattern 306, 307, 308 observed in the first time span contains enough information in 
order to classify the input pattern with adequate dependability. 

The first discrimination value I(T) is clearly formed (step 101) in the 
framework of the training for a first time span [0; T] (see Figure 1). 

In a further step (step 102), a second time span is formed by shortening the 
first time span: [0; T'], whereby T' < T applies. 

For the second time span [0; T'], a second discrimination value I(T') is 
formed in a further step (step 103) in the same way as described above for the first 
discrimination value I(T). 

The first discrimination value I(T) is compared to the second 
discrimination value I(T') (step 104). 
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When the second discrimination value I(T') is the same as the first 
discrimination value I(T), then a new second time span is formed (step 105) by 
shortening the second time span [0; T'], and the new second time span is considered 
to be the second time span (step 106). A second discrimination value I(T') is in turn 
formed (step 103) for the second time span of the new iteration. 

Clearly, this iterative method means that the time span wherein pulses 
generated by the pulsed neurons are taken into consideration for forming the output 
signal is shortened until the second discrimination value I(T') is unequal to the first 
discrimination value I(T). 

When the second discrimination value I(T') is smaller than the first 
discrimination value, then the neural network 301 is viewed as being an optimized 
neural network that was trained in the last preceding iteration wherein the second 
discrimination value I(T') was not smaller than the first discrimination value I(T) 
(step 107). 

The time span respectively taken into consideration is divided into discrete 
time sub-spans for which the only thing respectively determined is whether a neuron 
generated a pulse during the time sub-span or not. 

In this way, the calculating outlay needed for the training is considerably 

reduced. 

For further illustration, the principle is explained again on the basis of 

Fi gure 4 . 

Fi gure 4 shows two continuous processes pi and p2 that are formed by a 
set of continuous input signals Si and S2. Two sequences of input quantities, the 
input patterns, are present after a corresponding, above-described digitalization. The 
input patterns are supplied to the trained neural network 401 in an application phase, 
and a space-time encoding of the processes pi, p2 is clearly implemented on the basis 
of the time rows for the trained neural network 401 . 

On the basis of an output signal 402, the trained neural network 401 
indicates the kind of process the input pattern involves. The trained neural network 
401 exhibits the property that, first, the dependability of the optimization is optimized 
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and, second, a minimum plurality of time row values, i.e. a minimum second time 
span 403, is required in order to dependably implement the classification. 

A few alternatives to the above-described exemplary embodiment are 
present below: 

The plurality of inputs, of pulsed neurons as well as output signals is 
generally arbitrary. The plurality of different sequences of time row values is also 
arbitrary in the framework of the classification and in the framework of the training. 
An electroencephalogram analysis is thus possible for an arbitrary plurality of 
channels for characterizing tumors. 


The following publications have been cited in this document: 

[1] G. Deco and B. Schurmann, Information Transmission and Temporal 

Code in Central Spiking Neurons, Physical Review Letters, Vol. 79, No. 

23, pp. 4697-4700, December 1997 
5 [2] W. Gerstner, Time structure of the activity in neural network models, 

Physical Review E, Vol. 51, No. 1, pp. 738-758, January 1995 
[3] K. P. Unnikrishnan and K. P. Venugopal, Alopex: A Correlation-Based 

Learning Algorithm for Feedforward and Recurrent Neural Networks, pp. 

471-490, Neural Computation, Vol. 6, 1994 
10 [4] DE 195 31 967 C2 
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Patent Claims 

1 . Method for training a neural network that contains pulsed neurons, 

a) the neural network is trained such for a first time span ([0; T]) that a 
discrimination value is maximized, as a result whereof a maximum first 
discrimination value is formed; 

b) the discrimination value is formed dependent on pulses that are formed by 
the pulsed neurons within the first time span as well as on a training 
sequence of input quantities that are supplied to the neural network; 

c) the following steps are interactively implemented: 

- the first time span is shortened to form a second time span, 

~ a second discrimination value is formed for the second time span, 

- when the second discrimination value is the same as the first 
discrimination value, then a new iteration ensues with a new second time 
span that is formed by shortening the second time span of the preceding 
iteration, 

- otherwise, the method is ended and the trained neural network is the 
neural network of the last iteration wherein the second discrimination 
value is the same as the first discrimination value. 

2. Method according to claim 1, whereby an optimization method that is 
not gradient based is utilized for the maximization of the first discrimination value 
and/or of the second discrimination value. 

3. Method according to claim 2, whereby the optimization method is based 
on the ALOPEX method. 

4. Method according to one of the claims 1 through 3, whereby the first 
discrimination value I(T) satisfies the following rule: 


I(T) 


t (l) t (l) t (D t (2) t (2) t (2) 
.(n) t (n) t (n) M M M 


whereby 

s references the input quantities, 
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references a pulse that is generated by a pulsed neuron n at a time m 

within a time span [0, T], 
• k„ (n = 1, N) references a point in time at which the pulsed neuron n 

has generated the last pulse within the time span [0, T], 
5 • N references a plurality of pulsed neurons contained in the neural network. 

5. Method according to claim A, whereby the first discrimination value 
I(T) satisfies the following rule: 

I(T) = -Jpbut) • ln(p(out))d4 l) ...dtg...dt^ ) + 

+ fp,J^a»^0) ■ X I {p( 0 ut| s 6))) d t< 1 )...dt«... d tW 


with 


p(out) = £ Pjp(°ut|s^)} , 


whereby 

timej, 

pj references a probability that the input quantity s^ J ^ is applied to the 
neural network at a pint in time j, 
• p(out | ^) references a conditioned probability that a pulse is generated 

by a pulsed neuron in the neural network under the condition that the input 
quantity ^ is applied to the neural network at a point in timej. 

6. Method according to one of the claims 1 through 5, whereby the 
training sequence of inputs quantities are [sic] measured physical signals. 

7. Method according to claim 6, whereby the training sequence of input 
quantities are [sic] signals of an electroencephalogram. 
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8. Method for the classification of a sequence of input quantities upon 
employment of a neural network that contains pulsed neurons and was trained 
according to the following steps: 

a) the neural network is trained such for a first time span that a 
discrimination value is maximized, as a result whereof a maximum first 
discrimination value is formed; 

b) the discrimination value is formed dependent on pulses that are formed by 
the pulsed neurons within the first time span as well as on a training 
sequence of input quantities that are supplied to the neural network; 

c) the following steps are interactively implemented: 

— the first time span is shortened to form a second time span, 

— a second discrimination value is formed for the second time span, 
~ when the second discrimination value is the same as the first 
discrimination value, then a new iteration ensues with a new second time 
span that is formed by shortening the second time span of the preceding 
iteration, 

~ otherwise, the method is ended and the trained neural network is the 
neural network of the last iteration wherein the second discrimination 
value is the same as the first discrimination value, 

-- whereby the sequence of input quantities is supplied to the neural network; 

-- whereby a classification signal is formed that indicates what kind of sequence of 

input quantities the supplied sequence is. 

9. Method according to claim 9, whereby the training sequence of input 
quantities and the sequence of input quantities are measured physical signals. 

10. Method according to claim 9, whereby the training sequence of input 
quantities and the sequence of input quantities are measured signals of an 
electroencephalogram. 

1 1 . Neural network that contains pulsed neurons has been trained 
according to the following steps: 


22 

a) the neural network is trained such for a first time span that a 
discrimination value is maximized, as a result whereof a maximum first 
discrimination value is formed; 

b) the discrimination value is formed dependent on pulses that are formed by 
5 the pulsed neurons within the first time span as well as on a training 

sequence of input quantities that are supplied to the neural network; 

c) the following steps are interactively implemented: 

— the first time span is shortened to form a second time span, 

— a second discrimination value is formed for the second time span, 
10 — when the second discrimination value is the same as the first 

discrimination value, then a new iteration ensues with a new second time 
span that is formed by shortening the second time span of the preceding 
iteration, 

— otherwise, the method is ended and the trained neural network is the 
15 neural network of the last iteration wherein the second discrimination 

value is the same as the first discrimination value. 

12. Neural network according to claim 10, utilized for the classification of 
a physical signal. 

13. Neural network according to claim 10, utilized for the classification of 
2 0 a physical signal. 

14. Arrangement for training a neural network that contains pulsed 
neurons comprises a processor that is configured such that the following steps can be 
implemented: 

a) the neural network is trained such for a first time span that a 

2 5 discrimination value is maximized, as a result whereof a maximum first 

discrimination value is formed; 

b) the discrimination value is formed dependent on pulses that are formed by 
the pulsed neurons within the first time span as well as on a training 
sequence of input quantities that are supplied to the neural network; 

3 0 c) the following steps are interactively implemented: 

— the first time span is shortened to form a second time span, 
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- a second discrimination value is formed for the second time span, 
~ when the second discrimination value is the same as the first 
discrimination value, then a new iteration ensues with a new second time 
span that is formed by shortening the second time span of the preceding 

5 iteration, 

- otherwise, the method is ended and the trained neural network is the 
neural network of the last iteration wherein the second discrimination 
value is the same as the first discrimination value. 

15. Arrangement according to claim 14, utilized for the classification of a 
10 physical signal. 

16. Arrangement according to claim 14, utilized for the classification of a 
signal of an electroencephalogram. 
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Abstract 

METHOD FOR TRAINING A NEURAL NETWORK, METHOD FOR THE 
CLASSIFICATION OF A SEQUENCE OF INPUT QUANTITIES UPON 
EMPLOYMENT OF A NEURAL NETWORK, NEURAL NETWORK AND 
5 ARRANGEMENT FOR THE TRAINING OF A NEURAL NETWORK 
For a first time span, the neural network is trained such that a 
discrimination value is maximized, whereby the discrimination values is dependent on 
pulses that are formed by pulsed neurons within the first time span. Iteratively, the 
first time span is shortened and a second discrimination value is formed until the 
1 0 second discrimination value is smaller than the maximum discrimination value. The 
trained neural network is the neural network of the last iteration wherein the second 
discrimination value is equal to the maximum discrimination value. 
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(57) Abstract 

The neural network is trained for a first period in such 
a way that a discrimination value is maximized, wherein the 
discrimination value is dependent on impulses formed by the 
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(57) Zusammenfassung 
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der Unterscheidungswert abhangig ist von Impulsen, die von den 
gepulsten Neuronen innerhalb des ersten Zeitraums gebildet wer- 
den. Iterativ wird der erste Zeitraum so lange verkurzt und fur 
den zweiten Zeitraum ein zweiter Unterscheidungswert gebildet, 
bis der zweite Unterscheidungswert kleiner ist als der maximale 
Unterscheidungswert. Das trainierte neuronale Netz ist das neu- 
ronale Netz der letzten Iteration, bei der der zweite Unterschei- 
dungswert gleich dem maximalen Unterscheidungswert ist. 
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Ich beanspruche hiermit gemass Absatz 35 der Zivil- 
prozessordnung der Vereinigten Staaten, Paragraph 
120, den Vorzug aller unten aufgefuhrten Anmel- 
dungen und falls der Gegenstand aus jedem An- 
spruch dieser Anmeldung nicht in einer fruheren ame- 
rikanischen Patentanmeldung laut dem ersten Para- 
graphen des Absatzes 35 der ZivilprozelSordnung der 
Vereinigten Staaten, Paragraph 122 offenbart ist, 
erkenne ich gemass Absatz 37, Bundesgesetzbuch, 
Paragraph 1.56(a) meme Pflicht zur Offenbarung von 
Informationen an, die zwischen dem Anmeldedatum 
der fruheren Anmeldung und dem nationalen oder 
PCT internationalen Anmeldedatum dieser Anmel- 
dung bekannt geworden sind. 


I hereby claim the benefit under Title 35. United Sta- 
tes Code. §120 of any United States application(s) 
listed below and, insofar as the subject matter of each 
of the claims of this application is not disclosed in the 
prior United States application in the manner provided 
by the first paragraph of Title 35, United States Code, 
§122, I acknowledge the duty to disclose material 
information as defined in Title 37, Code of Federal 
Regulations, §1. 56(a) which occured between the 
filing date of the prior application and the national or 
PCT international filing date of this application. 


(Status) 

(patentierl, anhangig, 
aufgegeben) 


(Status) 

(patentiert, anhangig, 
aufgeben) 


(Status) 

(patented, pending, 
abandoned) 


Ich erklare hiermit, dass alle von mir in der vorliegen- 
den Erklarung gemachten Angaben nach meinem 
besten Wissen und Gewissen der vollen Wahrheit 
entsprechen, und dass ich diese eidesstattliche Erkla- 
rung in Kenntnis dessen abgebe, dass wissentlich und 
vorsatzlich falsche Angaben gemass Paragraph 1001, 
Absatz 18 der Zivilprozessordnung der Vereinigten 
Staaten von Amerika mit Geldstrafe belegt und/oder 
Gefangnis bestraft werden koennen, und dass derartig 
wissentlich und vorsatzlich falsche Angaben die Gul- 
tigkeit der vorliegenden Patentanmeldung oder eines 
darauf erteilten Patentes gefahrden konnen. 


I hereby declare that all statements made herein of 
my own knowledge are true and that all statements 
made on information and belief are believed to be 
true, and further that these statements were made 
with the knowledge that willful false statements and 
the like so made are punishable by fine or imprison- 
ment, or both, under Section 1001 of Title 18 of the 
United States Code and that such willful false state- 
ments may jeopardize the validity of the application or 
any patent issued thereon. 
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Declaration and Power of Attorney For Patent Application 
Erklarung Fur Patentanmeldungen Mit Vollmacht 

German Language Declaration 


Als nachstehend benannter Erfinder erklare ich hiermit 
an Eides Statt: 


n Wohnsitz, meine Postanschrift, und meine 
Staatsangehorigkeit den im Nachstehenden nach 
Namen aufgefuhrten Angaben entsprechen, 


dass ich, nach bestem Wissen der ursprungliche, 
erste und alleinige Erfinder (falls nachstehend nur ein 
Name angegeben ist) oder ein ursprunglicher, erster 
und Miterfinder (falls nachstehend mehrere Namen 
aufgefuhrt sind) des Gegenstandes bin, fur den dieser 
Antrag gestellt wird und fur den ein Patent beantragt 
wird fur die Erfindung mit dem Titel- 


Verfahren zum Trainieren eines neuronalen 
Netzes, Verfahren zur Klassifikation einer 
Folge von Eingangsgroften unter Verwen- 


dung eines neuronalen Netzes, neuronales 
Netz und Anordnung zum Trainieren eines 
neuronalen Netzes 


deren Beschreibung 

(zutreffendes ankreuzen) 
El hier beigefugt ist. 
D am 


PCT international Anmeldung 

PCT Anmeldungsnummer 

eingereicht wurde und am _ 


abgeandert wurde (falls tatsachlich abgeandert). 


Ich bestatige hjermit, dass ich den Inhalt der obigen 
Patentanmeldung einschliesslich der Anspruche 
durchgesehen und verstanden habe, die eventuell 
durch einen Zusatzantrag wie oben erwahnt abgean- 
dert wurde. 


Ich erkenne meine Pflicht zur Offenbarung irgendwel- 
cher Informationen, die fur die Prufung der vorliegen- 
den Anmeldung in Einklang mit Absatz 37, Bundes- 
gesetzbuch, Paragraph 1.56(a) von Wichtigkeit sind, 


Ich beanspruche hiermit auslandische Prioritatsvor- 
teile gemass Abschnitt 35 der Zivilprozessordnung der 
Vereinigten Staaten, Paragraph 119 aller unten ange- 
gebenen Auslandsanmeldungen fur ein Patent oder 
eine Erfindersurkunde, und habe auch alle Auslands- 
anmeldungen fur em Patent oder eine Erfindersurkun- 
de nachstehend gekennzeichnet, die ein Anmelde- 
datum haben, das vor dem Anmeldedatum der An- 
meldung liegt, fur die Prioritat beansprucht wird. 


As a below named inventor, I hereby declare that: 


My residence, post office address and citizenship a 
as stated below next to my name, 


I believe I am the original, first and sole inventor (if 
only one name is listed below) or an original, first and 
joint inventor (if plural names are listed below) of the 
subject matter which is claimed and for which a patent 
is sought on the invention entitled 


the specification of which 
(check one) 

□ is attached hereto. 

□ was filed on 


PCT international application 
PCT Application No 


and was amended on 


I hereby state that I have reviewed and understand the 
contents of the above identified specification, inclu- 
ding the claims as amended by any amendment refer- 
red to above. 


I acknowledge the duty to disclose information which 
is material to the examination of this application in 
accordance with Title 37, Code of Federal Regula- 
tions, §1. 56(a). 


I hereby claim foreign priority benefits under Title 35, 
United States Code, §119 of any foreign application(s) 
for patent or inventor's certificate listed below and 
have also identified below any foreign application for 
patent or inventor's certificate having a filing date 
before that of the application on which priority is clai- 
med: 
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